[dotnet-trace] Add collect-linux verb #5570

mdh1418 · 2025-09-17T21:50:29Z

Following the addition of emitting native runtime and custom EventSource events as user_events through dotnet/runtime#115265 and the public release of https://github.com/microsoft/one-collect which supports collecting both .NET user_events and Linux perf events into a single .nettrace file, dotnet-trace will support a new verb, collect-linux, that wraps around record-trace.

This PR does the following:

Adds collect-linux verb and serializes a subset of dotnet-trace collect options in addition to a collect-linux specific --perf-events option into record-trace args. (see [Diagnostics][dotnet-trace] Add collect-linux verb docs#47894 for overarching details)
Adds record-trace dynamic library to dotnet-trace
Updates existing profiles (cpu-sampling -> dotnet-common + dotnet-sampled-thread-time) and adds collect-linux specific profiles
Updates list-profiles verb with revamped profiles + multiline description formatting
Refactors EventPipeProvider composition logic (MergeProfileAndProviders + ToProviders -> ComputeProviderConfig) and rename Extensions.cs -> ProviderUtils.cs
Revamp EventPipeProvider composition tests (ProviderParsing.cs -> ProviderCompositionTests.cs)
Various cleanup: Update CLREventKeywords + Update logging + refactor collect logic + expand dotnet-trace common options

Testing

dotnet-trace collect-linux

On Linux

collect-linux

collect-linux --help

$ ./dotnet-trace collect-linux -h
Description:
  Collects diagnostic traces using perf_events, a Linux OS technology. collect-linux requires admin privileges to capture kernel- and user-mode events, and by default, captures events from
  all processes. This Linux-only command includes the same .NET events as dotnet-trace collect, and it uses the kernel’s user_events mechanism to emit .NET events as perf events, enabling
  unification of user-space .NET events with kernel-space system events.

Usage:
  dotnet-trace collect-linux [options]

Options:
  --providers      A comma delimited list of EventPipe providers to be enabled. This is in the form 'Provider[,Provider]',where Provider is in the form:
                   'KnownProviderName[:[Flags][:[Level][:[KeyValueArgs]]]]', and KeyValueArgs is in the form: '[key1=value1][;key2=value2]'.  Values in KeyValueArgs that contain ';' or '='
                   characters need to be surrounded by '"', e.g., FilterAndPayloadSpecs="MyProvider/MyEvent:-Prop1=Prop1;Prop2=Prop2.A.B;".  Depending on your shell, you may need to escape
                   the '"' characters and/or surround the entire provider specification in quotes, e.g., --providers
                   'KnownProviderName:0x1:1:FilterSpec=\"KnownProviderName/EventName:-Prop1=Prop1;Prop2=Prop2.A.B;\"'. These providers are in addition to any providers implied by the
                   --profile argument. If there is any discrepancy for a particular provider, the configuration here takes precedence over the implicit configuration from the profile.  See
                   documentation for examples.
  --clreventlevel  Verbosity of CLR events to be emitted.
  --clrevents      List of CLR runtime events to emit.
  --perf-events    Comma-separated list of perf events (e.g. syscalls:sys_enter_execve,sched:sched_switch).
  --profile        A named, pre-defined set of provider configurations for common tracing scenarios. You can specify multiple profiles as a comma-separated list. When multiple profiles are
                   specified, the providers and settings are combined (union), and duplicates are ignored.
  -o, --output     The output path for the collected trace data. If not specified it defaults to '<appname>_<yyyyMMdd>_<HHmmss>.nettrace', e.g., 'myapp_20210315_111514.nettrace'. [default:
                   default]
  --duration       When specified, will trace for the given timespan and then automatically stop the trace. Provided in the form of dd:hh:mm:ss.
  -?, -h, --help   Show help and usage information

`collect-linux` without elevated privileges

$ ./dotnet-trace collect-linux
==========================================================================================
The collect-linux verb is in preview. Some usage scenarios may not yet be supported,
and some trace parsers may not yet support NetTrace V6. For any bugs or unexpected
behaviors, please open an issue at https://github.com/dotnet/diagnostics/issues.
==========================================================================================
No providers, profiles, ClrEvents, or PerfEvents were specified, defaulting to trace profiles 'dotnet-common' + 'cpu-sampling'.

Provider Name                           Keywords            Level               Enabled By
Microsoft-Windows-DotNETRuntime         0x000000100003801D  Informational(4)    --profile

Linux Perf Events                                                               Enabled By
cpu-sampling                                                                    --profile

Output File    : /home/mihw/repo/diagnostics/trace_20251022_152234.nettrace

Error: Tracefs is not accessible: Permission denied (os error 13)

`collect-linux` with elevated privileges

$ sudo ./dotnet-trace collect-linux
==========================================================================================
The collect-linux verb is in preview. Some usage scenarios may not yet be supported,
and some trace parsers may not yet support NetTrace V6. For any bugs or unexpected
behaviors, please open an issue at https://github.com/dotnet/diagnostics/issues.
==========================================================================================
No providers, profiles, ClrEvents, or PerfEvents were specified, defaulting to trace profiles 'dotnet-common' + 'cpu-sampling'.

Provider Name                           Keywords            Level               Enabled By
Microsoft-Windows-DotNETRuntime         0x000000100003801D  Informational(4)    --profile

Linux Perf Events                                                               Enabled By
cpu-sampling                                                                    --profile

Output File    : /home/mihw/repo/diagnostics/trace_20251022_152124.nettrace

[00:00:00:21]   Recording trace.
Press <Enter> or <Ctrl-C> to exit...
Recording stopped.
Resolving symbols.
Finished recording trace.
Trace written to /home/mihw/repo/diagnostics/trace_20251022_152124.nettrace

On Windows (and I presume other non-Linux OS):

`collect-linux`

.\artifacts\bin\dotnet-trace\Debug\net8.0\dotnet-trace.exe collect-linux
The collect-linux command is only supported on Linux.

dotnet-trace list-profiles

`list-profiles`

dotnet-trace profiles:
        dotnet-common                        - Lightweight .NET runtime diagnostics designed to stay low overhead.
                                               Includes GC, AssemblyLoader, Loader, JIT, Exceptions, Threading, JittedMethodILToNativeMap, and Compilation events
                                               Equivalent to --providers "Microsoft-Windows-DotNETRuntime:0x100003801D:4".
        dotnet-sampled-thread-time (collect) - Samples .NET thread stacks (~100 Hz) toestimate how much wall clock time code is using.
        gc-verbose                           - Tracks GC collections and samples object allocations.
        gc-collect                           - Tracks GC collections only at very low overhead.
        database                             - Captures ADO.NET and Entity Framework database commands
        cpu-sampling (collect-linux)         - Kernel CPU sampling events for measuring CPU usage.
        thread-time (collect-linux)          - Kernel thread context switch events for measuring CPU usage and wall clock time

src/Tools/dotnet-trace/CommandLine/Commands/CollectCommand.cs

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs

src/Tools/dotnet-trace/CommandLine/Options/CommonOptions.cs

src/Tools/dotnet-trace/ProviderUtils.cs

src/tests/dotnet-trace/ProfileProviderMerging.cs

src/Tools/dotnet-trace/ProviderUtils.cs

Extensions houses provider composition helpers, instead of literal extension methods for strings. Centralize all provider parsing + composition logic and keep Profile as a data container.

src/Tools/dotnet-trace/dotnet-trace.csproj

noahfalk

This is looking pretty good. Aside from some small comments inline a few broader things:

Historically we relied on manual testing to keep these tools operating well but now that we have less time available from the testers I think we need to improve our automated testing. Partly this is to ensure we aren't inadvertently changing the original 'collect' verb and partly to ensure going forward the 'collect-linux' behavior doesn't regress either. I think the best way to do this would be:

Open a new PR that we'll check in first containing some basic tests of the existing collect verb.
Commit this PR 2nd and all the tests in the 1st PR should continue to pass. This ensures we didn't change anything unintended.

To do the testing we probably need to create some small interface shims. We already have an IConsole interface defined that could be moved to the shared Common folder. We could also create a small interface around the DiagnosticsClient.EventPipeCollect() API so that a test can return some dummy data in a stream instead. dotnet-counters has some example tests that show how we can run some code and then confirm the console output is what we expect. In this case I imagine we'd be running the Collect() function and giving some chosen input arguments.

I think there is a bit more adjustment to be done on some of the output text, but it should be fine to get this one in first, then tweak afterwards in some 3rd PR.

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs

src/Tools/dotnet-trace/ProviderUtils.cs

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs

src/tests/dotnet-trace/CollectCommandFunctionalTests.cs

noahfalk · 2025-10-08T06:42:28Z

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs

        private static Stopwatch s_stopwatch = new();
-        private static LineRewriter s_rewriter = new() { LineToClear = Console.CursorTop - 1 };
+        private static LineRewriter s_rewriter;
        private static bool s_printingStatus;


If testing caused multiple instances of CollectLinuxCommandHandler to run in parallel, would we want those instances to be sharing any of these static fields? I'm guessing we'd want all these fields to be instanced too.

Changed to non-static. From some learning, I'm seeing that parallelization occurs at the test class level, so it doesn't seem like the theory members would run in parallel.

src/tests/dotnet-trace/CollectLinuxCommandFunctionalTests.cs

noahfalk · 2025-10-08T06:50:31Z

src/tests/dotnet-trace/CollectLinuxCommandFunctionalTests.cs

+                    LinuxHeader,
+                    LinuxProfile("cpu-sampling"),
+                    ""
+                }


Shouldn't there be some output about where the test output is being written to and telling the user to press Enter to stop the trace?

I added the output file to Collect-linux, thanks for the catch. As per the press enter/ctrl+c message, normally that is part of the status printing.

For collect-linux, I added the line rewriting/status printing to the callback, so to mock that for the tests, extended the RecordTraceInvoker test seam to invoke the callback once with the progress output type, causing the status+ Press Enter/Ctrl+C to emit once.

noahfalk

A few comments inline around the error handling, but if you want to handle that in a separate PR I'm fine with it. In terms of checking this in, lets make sure Juan gets the release out before this gets checked in.

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs

src/tests/dotnet-trace/CollectCommandFunctionalTests.cs

src/Tools/dotnet-trace/CommandLine/Commands/CollectCommand.cs

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs

…eption To swallow stacktraces from expected invalid cli option combinations without swallowing stacktraces from other ArgumentExceptions thrown, swap to throwing the tools common custom CommandLineErrorException.

noahfalk

I've got one last suggestion to adjust the preview banner text, but otherwise I think we are looking good 👍

noahfalk · 2025-10-23T01:04:46Z

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs

+            }
+
+            Console.WriteLine("==========================================================================================");
+            Console.WriteLine("The collect-linux verb is in preview. Some usage scenarios may not yet be supported,");


I doubt the average user will know what "NetTrace V6" means :) They probably recognize the file suffix but not the "V6". I'd also suggest we can make the message a little useful by telling them what will work and pointing them at the docs. To keep the message from getting too long we can put bug filing info as part of a longer message we include in the docs. How about this:

"The collect-linux verb is a new preview feature and relies on an updated version of the .nettrace file format. The latest PerfView release supports these trace files but other ways of using the trace file may not work yet. See the docs at https://learn.microsoft.com/dotnet/core/diagnostics/dotnet-trace for more details."

Added, I've been separating the lines to where each line is just under 90 characters, which is the width of
Provider Name Keywords Level Enabled By
so slightly adjusted

mdh1418 · 2025-10-23T15:02:10Z

The test failure is #2541

mdh1418 requested review from hoyosjs, lateralusX and noahfalk September 17, 2025 21:50

noahfalk reviewed Sep 19, 2025

View reviewed changes

mdh1418 force-pushed the dotnet_trace_collect_linux branch from c09b095 to 0989d21 Compare September 19, 2025 05:45

mdh1418 added 12 commits September 19, 2025 11:08

[DotnetTrace] Update CLREventKeywords

536cd50

[DotnetTrace] Move MergeProfileAndProviders to Extensions

49a875b

Extensions houses provider composition helpers, instead of literal extension methods for strings. Centralize all provider parsing + composition logic and keep Profile as a data container.

[DotnetTrace] Rename Extensions to ProviderUtils

b4787fc

[DotnetTrace] Add provider unifying helper

c921869

[DotnetTrace] Move shared options to CommonOptions

e1c15c7

[DotnetTrace] Add collect-linux skeleton

b4112b5

[DotnetTrace][CollectLinux] Start record-trace

e5f3975

[DotnetTrace][CollectLinux] Build record-trace args

677e54e

[DotnetTrace] Update profiles

31186a8

[DotnetTrace] Update collect to new provider unifier

44593bb

[DotnetCounters] Remove Extensions reference

b190d73

[DotnetTrace] Update tests

9f1ef6d

mdh1418 force-pushed the dotnet_trace_collect_linux branch 2 times, most recently from 76e65e1 to 61b2569 Compare September 19, 2025 18:32

Address Feedback

c8e53ea

mdh1418 force-pushed the dotnet_trace_collect_linux branch from 61b2569 to c8e53ea Compare September 19, 2025 18:56

[DotnetTrace] Print profile effects and clrevents ignore warning

590a203

mdh1418 marked this pull request as ready for review September 19, 2025 19:43

mdh1418 requested a review from a team as a code owner September 19, 2025 19:43

mdh1418 added the DO NOT MERGE do not merge this PR label Sep 19, 2025

mdh1418 changed the title ~~[dotnet-trace] Add collect-linux verb~~ [NO-MERGE][dotnet-trace] Add collect-linux verb Sep 19, 2025

mdh1418 commented Sep 19, 2025

View reviewed changes

src/Tools/dotnet-trace/dotnet-trace.csproj Outdated Show resolved Hide resolved

[DotnetTrace] Print PerfEvents and include in default condition

8e2453f

noahfalk reviewed Sep 24, 2025

View reviewed changes

[DotnetTrace] Update OneCollect package with FFI

573d769

noahfalk reviewed Oct 8, 2025

View reviewed changes

mdh1418 added 2 commits October 8, 2025 18:06

Adjust functional tests and add failure cases

208086f

[DotnetTrace][CollectLinux] Add ProgressStatus and output file to tests

40c5905

mdh1418 force-pushed the dotnet_trace_collect_linux branch from eff3e5f to 40c5905 Compare October 8, 2025 18:10

noahfalk previously approved these changes Oct 14, 2025

View reviewed changes

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs Outdated Show resolved Hide resolved

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs Show resolved Hide resolved

src/tests/dotnet-trace/CollectCommandFunctionalTests.cs Outdated Show resolved Hide resolved

mdh1418 added 2 commits October 17, 2025 20:15

Cleanup variable names

32f01dc

Handle invalid provider config exception

25d5559

mdh1418 dismissed noahfalk’s stale review via 25d5559 October 17, 2025 20:18

mdh1418 added the DO NOT MERGE do not merge this PR label Oct 17, 2025

mdh1418 changed the title ~~[dotnet-trace] Add collect-linux verb~~ [NO MERGE][dotnet-trace] Add collect-linux verb Oct 17, 2025

noahfalk reviewed Oct 17, 2025

View reviewed changes

src/Tools/dotnet-trace/CommandLine/Commands/CollectCommand.cs Outdated Show resolved Hide resolved

src/Tools/dotnet-trace/CommandLine/Commands/CollectLinuxCommand.cs Outdated Show resolved Hide resolved

mdh1418 added 3 commits October 20, 2025 23:44

Add Preview message

6f88dc7

Fix remnant tests expecting ArgumentException

3a4b1f2

mdh1418 changed the title ~~[NO MERGE][dotnet-trace] Add collect-linux verb~~ [dotnet-trace] Add collect-linux verb Oct 22, 2025

mdh1418 removed the DO NOT MERGE do not merge this PR label Oct 22, 2025

mdh1418 added 3 commits October 22, 2025 15:16

Update LineToRewrite prior to rewriting

fc05a0a

Add banner to preview message

2126a8d

Bump RecordTrace to support ActivityIDs

119f101

mdh1418 mentioned this pull request Oct 22, 2025

[dotnet-trace] Convert and Report not compatible with collect-linux traces #5610

Open

noahfalk previously approved these changes Oct 23, 2025

View reviewed changes

Improve preview banner message

e9b0aa8

mdh1418 dismissed noahfalk’s stale review via e9b0aa8 October 23, 2025 03:10

noahfalk approved these changes Oct 23, 2025

View reviewed changes

mdh1418 merged commit d4961a0 into dotnet:main Oct 23, 2025
19 checks passed

This was referenced Oct 24, 2025

[release/10.0.1xx] Source code updates from dotnet/diagnostics dotnet/dotnet#3057

Merged

[main] Source code updates from dotnet/diagnostics dotnet/dotnet#3081

Merged

noahfalk mentioned this pull request Oct 29, 2025

Enable command utils console redirection #5622

Open

mdh1418 mentioned this pull request Nov 3, 2025

[UserEvents] Add end-to-end runtime test dotnet/runtime#121316

Open

[dotnet-trace] Add collect-linux verb #5570

[dotnet-trace] Add collect-linux verb #5570

Uh oh!

Conversation

mdh1418 commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing

dotnet-trace collect-linux

dotnet-trace list-profiles

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

noahfalk left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

noahfalk Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

mdh1418 Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

noahfalk Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

mdh1418 Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

noahfalk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

noahfalk left a comment

Choose a reason for hiding this comment

Uh oh!

noahfalk Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

mdh1418 Oct 23, 2025

Choose a reason for hiding this comment

Uh oh!

mdh1418 commented Oct 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mdh1418 commented Sep 17, 2025 •

edited

Loading

noahfalk left a comment •

edited

Loading

mdh1418 Oct 8, 2025 •

edited

Loading

mdh1418 Oct 8, 2025 •

edited

Loading